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PERIODICITY PITCH AND RELATED AUDITORY PROCESS MODELS * 
J. C. RB. Licklider 


Abstract 


Schouten’s (34-38) residue and other phenomena call for extension of place 
theory to account for the role of periodicity in determining subjective pitch. 
This paper examines several mechanisms that might be used by the auditory 
system to recode the information carried by the time pattern of the output of 
the cochlear frequency analyzer. A distinction is made between mechanisms 
that involve ordered arrays of components and mechanisms that involve un- 
ordered arrays. The relation of periodicity mechanisms to ,,property filters” 
is examined, and periodicity pitch is discussed in relation to ,, sharpening” and 
binaural interaction. 


Introduction 


During the last 25 years, the study of hearing has advanced 35 millimeters. 
At the beginning of the period, it was known that the cochlea performed a 
mechanical frequency analysis upon vibratory signals delivered to it through 
the middle ear or temporal bone, but neither the general structure of the 
process of analysis nor the exact nature of the product of analysis was clear. 
The analysis was the subject of speculations and conjectures. Now, largely 
as a result of Békésy’s (2, 3) celebrated observations, the mechanical ana- 
lysis is well enough understood that the speculations and conjectures have 
been replaced by graphs, charts, and computer models (9, 10). Investigation 
has shifted to later stages of the auditory process. 


The process through which neurons of the auditory nerves are excited 
seems to be understood now perhaps a little better than cochlear mechanical 
analysis was understood in 1937. Basic features of the responses of neurons 
in the auditory nerve, and of neurons and aggregates of neurons in auditory 
centers of the brain, have been observed. (15, 41) However, no one now has 
knowledge much more advanced than ,,hunches’”’ concerning the overall plan 
and products of the neural auditory process. The action of the auditory ner- 
vous system is therefore now the subject of approximately the same level 
of speculation and conjecture as the mechanical analysis was before Békésy’s 
observations. 


*) Preparation of this paper was supported in part by the Air Force Office of Scien- 
tific Research under Contract AF 49(638)-355, AFOSR 2681. 
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The purpose of this paper is to summarize recent ideas on one small but 
perhaps crucial topic, the role of periodicity in the perception of pitch. This 
topic has had a central position in audition, off and on, ever since the dis- 
cussions of Seebeck (35), Ohm (31,32), and Helmhol:z (20). Interest in the 
topic is now particularly great because there are enough facts to make it 
clear that periodicity plays a significant role in pitch perception, yet not 
enough facts to define the role precisely or to indicate how it is played. 

By using the words ,,speculation” and ,,conjecture’, | mean to convey not 
a negative evaluation of the ideas to be summarized but a qualification: a de- 
claration of tentativeness and a warning of probing beyond established facts. 
The word ,,theory’’, it seems to me, is also appropriate, but only after a dis- 
tinction is made between the present hypothetical, heuristic kind of theory 
intended to facilitate discovery (and thereby almost surely get itself dis- 
proved), and the rarer synthetic, consolidating kind of theory, intended to 
organize established facts and facilitate their incorporation into the body of 
knowledge. 


PERIODICITY AND FOURIER FREQUENCY 


The period of a sinusoidal wave is so well known to be the reciprocal of the 
frequency of the sinusoidal wave, and we are so accustomed to thinking 
in terms of frequency, that periodicity may seem at first thought to be only 
the other face of a familiar coin. Usually it is. A necessity for making a further 
distinction between periodicity and frequency — further than that ,,the period 
is the reciprocal of the frequency’’ — arises, however, when we consider 
certain compound (nonsinusoidal) waveforms. The waveform in Fig. 1A, for 
example, is periodic with period T, yet it contains no power at the Fourier 
frequency f = 1/T. Thus the wave has a definite periodicity not corresponding 
to any frequency energetically present in the spectrum. Clearly, the perio- 
dicity suggests things that are not suggested directly by the frequency com- 
position per se. Even to a beginning student (or perhaps particularly to a be- 
ginning student), the waveform suggests that the stimulus will give rise to a 
sensation similar in pitch to that produced by a 200 Hz tone, whereas the 
spectrum seems to suggest that the pitch will be high. 

The waveform in Fig. 1B is not itself periodic, but its envelope is periodic 
with period T. The waveform of 1B itself has no power at the Fourier fre- 
quency f — 1/T. It is of interest to deliver to the ear waveforms such as those 
of 1A and 1B and to examine the subjective experiences to which they give 
rise. 


»PLACE PITCH” AND ,,PERIODICITY PITCH” 

Place Theory 

Because the ,,place theory” has been dominant for so long a time, and be- 
cause periodicity mechanisms are subsequent (if not secondary) to place 
mechanisms in the auditory process, we need a fairly definite statement 
of the place theory of pitch perception to use as a point of departure. The 
statements made by Helmholtz (20) and other classical place theorists seem 
to me not to be entirely appropriate for that purpose; they are too far re- 
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Fig. 1. Periodic waveforms illustrating distinction between frequency and the re- 
ciprocal of period. The waveform of A is periodic with period T, but (as the spectrum 
below it shows) contains no power at freqency f = 1/T. The envelope of the waveform 
of B (but not the waveform itself) is periodic with period T. The spectrum correspond- 
ing to the waveform has no power at f = 1/T; the spectrum corresponding to the en- 
velope has. 
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moved from the modern context. Perhaps it would be helpful for the state- 
ment to mention specifically the cochiear partition (or the basilar membrane) 
and to make less specific reference to a ,,perceptual’’ region of the brain: 

1. To each frequency along the frequency scale of the running spectrum 
of the stimulus there corresponds a point along the length of the basilar 
membrane. The mechanical action of the cochlea distributes each frequency 
component of the running spectrum over an interval about the corresponding 
point. The intervals are broad, especially for low-frequency components. The 
transformation is approximately linear for stimuli of low intensity. 

2. To each point along the basilar membrane, there corresponds a point 
along a continuum of neural tissue in a ,,perceptual’’ region of the nervous 
system. The nervous system maps the basilar membrane onto the perceptual 
continuum in such a way as to preserve ordinal relations. The neural trans- 
formation is a ,,projection’: local facilitory and inhibitory interactions may 
modify (particularly, sharpen) the distribution of activity, but remote inter- 
actions are weak relative to local ones. 

3. To each point along the continuum in the perceptual region of the ner- 
vous system there corresponds a point along a continuum of subjective 
pitch. In instances in which activity is concentrated about one or a few points 
on the neural continuum, subjective pitches arise at (or about) corresponding 
points on the pitch continuum. In instances in which activity is uniformly 
distributed over the neural tissue, either no subjective pitch arises, or a sub- 
jective pitch arises that depends upon some weighting of the activities at 
various points along the neural continuum. 

4. An influence called ,,attention’’ may affect the coefficients (,,gains’’) 
of the projection channels and the strengths of the local interactions in the 
projection mechanism. It may also (at the discretion of the theorist) affect the 
coupling between the perceptual continuum in the nervous system and the 
subjective pitch continuum. 

5. Therefore, a single concentration of energy along the frequency scale 
of the running spectrum gives rise to a corresponding single pitch. If there 
are several or many concentrations of energy along the frequency scale, 
there may be several pitches or only one pitch, depending upon the states 
of the projection mechanism and attention. In any case, the pattern in pitch 
deviates from the pattern in frequency only in ways that can be attributed to 
facilitory and inhibitory interactions that are predominately local in a un- 
idimensional (but possibly multi-level) projection system. 

The foregoing formulation of place theory of course involves concepts 
that did not appear explicitly in the writings of the classical place theorists. 
It represents an attempt to extend the classical theory into the present con- 
text, to be specific where specificity is warranted and to be general (or vague) 
where neither the classical theory nor the current situation provides a basis 
for making a specific choice. 

In the formulation, there is no reference to the fine timing of events within 
the temporal resolution of the running spectrum. That is to say, place theory, 
as here stated, is indifferent to timing finer than about 0.1 sec. It therefore 
ignores the basis of anything that might be called ,,periodicity pitch’. Its 
pitch is ,,place pitch’ — pitch related to place of excitation or activation. 
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Phenomena Difficult to Explain Through Place Theory 


The present difficulties of ,pure” place theory are caused almost entirely 
by two phenomena, Schouten’s (34-38) residue phenomenon and Huggins’ 
(28, 8) binaural-pitch phenomenon. 

Since the residue phenomenon was elucidated in the preceding paper, 
| shall not discuss it in detail here. The essential points are: 

1. A low subjective pitch may be associated with the middle-or high-fre- 
quency components of a compound, line-spectrum stimulus, even when no 
corresponding frequency component is energetically present in the stimulus. 

2. The residue is distinguishable subjectively from the fundamental tone 
or difference tone that is introduced (if the sound pressure level is high) by 
nonlinear distortion in the middle ear or pre-neural cochlear mechanism (37). 
The residue may be heard simultaneously with the fundamental or difference 
tone. (37). 

3. The low pitch of the residue persists even though the low-frequency 
channels are saturated with random masking noise (27, 43). Thus, in direct 
contradiction of pure place theory, a low pitch may be heard through high- 
frequency cochlear channels. 

The chief counter-argument used in response to the difficulty presented by 
the residue has been that the ,,missing fundamental” or a difference-frequency 
component, though perhaps not present in the acoustic stimulus, is introduced 
by nonlinear distortion at some stage prior to the part of the auditory (co- 
chlear) process in which the various frequency components are distributed 
to their proper channels. This counter-argument was used by Helmholtz (20), 
Fletcher (12, 13), Hoogland (21), and many others. Unless counter-counter- 
evidence is at hand, the counter-argument is effective against point 1 (above). 
However, Schouten (37, 38) had, and has, good counter-counter-evidence. The 
counter-argument is ineffective against points 2 and 3 (above). In my opinion, 
the conclusion concerning low pitches without low frequencies should be that: 


a. Under some circumstances, the ,,missing fundamental’ (or difference 
tone) is reintroduced by nonlinear distortion, and the reintroduced component 
accounts for the (low) pitch reported. 

b. Under other circumstances, nonlinear distortion contributes, but the low 
pitch is due also in part to time-patierned activation of middle- or high-fre- 
quency channels. 

c. And under still other circumstances, perhaps only under the special 
circumstances of carefully planned experiments, all spurious contributions 
assignable to low-frequency channels of the cochlear output are accounted 
for or eliminated, and low pitch arises through high-frequency channels. 

Thus the conclusion at this point is not that the place-theory mechanism 
should be discarded, or that it never accounts for the facts. The conclusion 
at this point is only that there is one kind of observation that pure place 
theory (as formulated) is inadequate to explain, and that place theory there- 
fore needs to be extended. 

In the Huggins (28) phenomenon, or Huggins-Cramer phenomenon, since 
now it has been explored systematically by Cramer and Huggins (8), pitch 
arises from a binaural interaction. 
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The stimulus presented to the listener's left ear is a random noise with 
uniform spectrum. Alone, it has no very definite pitch. It sounds like ,,shh...”. 
The stimulus presented to the listener’s right ear is also a random noise with 
uniform spectrum. Alone, it sounds quite like the other. However, the two 
random noises have been derived from a common source. They are alike 
and in phase up to some frequency f,—Af and above f,+-Af. Between f.—Af 
and f,+Af, however, the two noises are unlike in respect of the phasing of 
their components. When the two noises are presented together, one to the left 
ear and the other to the right, the listener hears a faint tone, deep in a noisy 
background. He can report its pitch, which depends ufon f, and is appropriate 
to it. 

The phenomenon just described is limited to values of f, below about 1400 
cps and to values of Af worked out by Cramer and Huggins. For classical 
place theory, it poses an embarrassing problem, for the theory purports to 
define the physical and/or physiological correlates of subjective pitch, the 
theory says nothing about binaural interaction, and here is a controllable low 
pitch arising through binaural interaction from stimuli that monaurally are 
nothing but random noise. Note, however, that — like the residue — the Hug- 
gins-Cramer phenomenon calls not for a rejection but only for an extension 
of place theory. 


POSSIBLE MECHANISMS FOR PERIODICITY PITCH 


There appear to be two main approaches to the study of neural processes: 
(1) to examine the nervous system (with electrodes, dyes, etc.) and see what 
it suggests, and (2) to guess what objective the nervous system is trying to 
achieve, to consider various techniques for achieving that objective, and then 
to examine the nervous system and see whether it appears to be using any 
of those techniques. Most researchers use a mixed strategy, but | think 
Professor Békésy and many physiologists seem to favor approach (1), which 
gets immediately to the question at hand, whereas many engineers, psycho- 
logists, and psychophysicists prefer (2), which offers them an opportunity 
to bring their own special knowledge to bear upon the development of a 
theory before nature has a chance to prove the theory wrong. In any event, 
we are involved here in approach (2). 

The process that transforms the pattern of the mechanical vibration of the 
basilar membrane into a pattern of discharge of neurons in the auditory nerve 
has at least one feature of great significance in connection with periodicity 
pitch. The excitation of neurons is inherently a nonlinear process similar to 
rectification. It is sensitive, therefore, to frequencies not energetically present 
in the vibration of the membrane, and it converts ,,missing fundamentals”’ into 
frequencies of neural discharge just as well as it does ,,present fundamen- 
tals”. The crucial point is, of course, that it sets up a train of discharges at 
the frequency of a missing 200-Hz fundamental, for example, in bundles of 
neurons that constitute middle- or high-frequency channels, and not in the 
bundles that would be excited by energetic sinusoidal vibrations near 200 Hz. 
Our problem, therefore, is to extend or modify the place theory in such a 
way that trains of impulses at 200 cps in middle- and high-frequency channels 
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of the auditory nerve can give rise to, or contribute to, a low subjective pitch. 

At the present time, neural signals are usually thought of either as (a) trains 
of nerve impulses in axons or axon bundles or as (b) complexes of ,,spike” 
and ,,local’’ potentials (together with associated chemical activity) in the cell 
bodies, dendrites, and axons of neurons in central nuclei plus the ,,slow’’ 
potentials that are recorded throughout central nuclei. Neurons themselves 
are recognized to be quite complex and diverse, and neuronal networks, ex- 
tremely complex and diverse. There is no dearth of components with which 
to synthesize hypothetical processing systems. ; 

Despite the richness of possibilities for detailed configuration, there seem 
to be only a few basic operations for discrimination of periodicity. Any one 
of them could be applied in a thousand particular ways to the task of dis- 
criminating the periodicities in the cochlear output, but fortunately it may not 
be necessary to list the particular manifestations in order to sensitize oneself 
to evidence of one of them in experimental data. 


Time-Domain Representation 


The first general possibility is that periodicity remains encoded in the time 
domain until the signals leave the auditory system. This amounts to saying 
that the ultimate neural variable underlying subjective pitch is frequency of 
neuronal! discharge. It does not entirely avoid the problem of transforming 
the message out of the time domain, for such a transformation must be made 
before there can be a selective verbal (or other motor) response. 

In any event, this first general possibility has two subpossibilities: (1) that 
periodicity is simply preserved, more or less accurately, but without reduct- 
ion of impulse repetition rate, and (2) that pulse-count division or pulse ,,fre- 
quency scaling” is employed. The volley mechanism of Wever and Bray (45) 
is of course the mechanism of choice for (1), but the measurements of Gold- 
stein and Kiang (19) underscore the conclusion that the highest centers of the 
auditory system do not “follow frequency” beyond about 200 Hz. In as much 
as the ,average response” technique used by Goldstein and Kiang is re- 
sponsive tot frequency-scaled as well as unscaled impulsive responses, so long 
as they are time locked, their failure to find time-locked cortical responses to 
high-frequency stimulus pulse trains argues against simple frequency-scaling 
hypotheses, also. My feeling, therefore, continues to be that ,,time domain all 
the way” is not promising, either as a substitute for or as an extension of 
place theory. 

Accumulating evidence that the cortex is not essential for frequency dis- 
crimination suggests that we keep open the hypothesis that, up to 1000 or 
even 2000 pulses per second, neural frequency or periodicity may be effec- 
tive at the thalamic level. We might look there, just as well as at lower levels, 
for a periodicity-to-place transformation. But that takes us out of the category 
of time-domain representations. 


Periodicity-To-Place Transformation 


The second general possibility is that the information carried in the periodicity 
or fine time structure of the cochlear output is transformed into the ,,place 
domain” at some stage, presumably a fairly early stage, of the auditory 
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process. Again there are two sub-possibilities: (1) that the transformation is 
one that maps neural frequency into a definite dimension of the neural tissue, 
a dimension that underlies or corresponds to subjective pitch, and (2) that 
the transformation merely allocates different frequencies to different neurons, 
the locations of which are significant, not in relation to any definite dimension 
or coordinate system, but only through interconnections with other neurons 
in a system that, insofar as macroscopic spatia! arrangement is concerned, 
appears more or less random. 

The task of a ,,periodicity-to-place”’ transformation system is to accept a 
single function of time and to produce a time-varying pattern in one or more 
spatial dimensions. In the input time function, two different levels or orders 
of time are significant, ,,gross’’ time and ,,fine’ time. The fine structure of 
the waveform may be regarded as undergoing slow variations, as varying in 
gross time. The changes of the output pattern must be slow changes; the out- 
put pattern must vary in gross time. Each momentary spatial pattern of the 
output signal must be a representation, in some form, of the fine structure 
in a corresponding segment of the input wave. Perhaps the most widely fa- 
miliar ,,periodicity-to-place” converter is extremely simple in principle: the 
cathode ray oscilloscope. 

Other more-or-less familiar systems for making transformations of the type 
discussed are: 


1. A filter bank — an ordered array of band-pass, low-pass, high-pass or 
other filters; 


2. A Fourier transformation system — an ordered array of correlators with 
reference signals supplied by a corresponding array of oscillators; 


3. An autocorrelation system — an ordered array of correlators with re- 
ference signals derived from the input signal through successive taps on a 
delay line; 

4. A power-serijes transformation system — an ordered array of correlators 
with reference signals supplied by direct-current, ramp, parabola, cubic, etc., 
generators. 


Note that the cathode-ray oscilloscope and the power-series transformation 
system have an intrinsic tendency to break the input wave up into segments 
and represent the segments successively as ,,frames’’. Filter banks, Fourier 
transformers, and autocorrelators, on the other hand, more-or-less naturally 
yield representations that develop continuously and progressively in gross 
time. Although the advantage is hard to evaluate, the continuous-gross-time 
feature favors the latter schemes for applications in auditory theory. 


Filter-Bank Model 


Because the excitation of nerve impulses is inherently nonlinear, a neuronal 
network would have to be rather complex to simulate closely the behavior 
of a linear wave filter. However, with quite simple arrangements of idealized 
neurons one can achieve interesting patterns of frequency selectivity. Is it not 
conceivable that the nervous system might use arrays of simple neural filters 
to transform patterns in the frequency or periodicity of auditory nerve dis- 
charges into place patterns? 
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Fig. 2. Frequency-selective neuronal network. a is the signal (schematized) at the 
input axon end-brush. B is the signal at the auxiliary axon end-brush. C is the signal 
at the cell body of the output neuron, which fires whenever ® and 6 ore simultane- 
ously active. In the first example, the output follows each discharge of the input except 


the first. In the second example, illustrating an effect analogous to anti-resonance, the 
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C suppresses response to multiples of the input frequency in the illustration, but not 
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Perhaps the simplest schema for a neural filter is shown in Fig. 2. A is the 
input neuron. B is an auxiliary neuron, C is the output neuron. We must as- 
sume that, when A discharges, an excitatory signal appears at each of its 
three terminals, that the two excitations applied to B are sufficient to make B 
discharge, and that, whereas two would be sufficient, the one excitation at 
C is insufficient to make C discharge. Let the delay introduced by B be tr. 
Then, if the period of the input pulse train (1) is approximately equal to 7, the 
spatial summation effect at C gives rise to an output pulse for each input 
pulse except the first. However, if the input train is not ,,in tune” with a na- 
tural resonance of the filter, the output is zero. Note that this filter has more 
than one ,,resonance” — it is more like a comb filter than like a band-pass 
filter. 

In the foregoing oversimplification, nothing was said about post-discharge 
refractoriness of the neurons, and no use was made of inhibition. Somewhat 
more complex arrangements of schematic neurons with those properties are 
capable of responding only to one band of input frequencies, of displaying 
build-up transients, of ,,ringing’, etc. The arrangement used in Fig. 3, for 
example, uses an inhibitory connection (blacked-in terminal) to suppress 
response to high-frequency input trains. The arrangement of Fig. 4, which 
doubtless would receive a low score if judged for cytoarchitectonic plausi- 
bility, illustrates the synthesis of a filter with an ,impulse response’ more 
nearly similar to the impulse response of a laboratory band-pass filter. All 
the synapses at B and C are assumed to produce faithful following of excit- 
ation. The terminals impinging upon D, on the other hand, contribute exci- 
tatory (B) and inhibitory (C) influences, and D responds if their sum is higher 
than some threshold value. The arrangement is therefore very sensitive to 
stimulation by pulses arriving at intervals near T, and therefore to pulse fre- 
quencies near F = 1/T. Although it might do so, it seems doubtful that the 
nervous system would need to go this far in approximation of laboratory fil- 
ter characteristics. 

It is easy to imagine a neural filter bank consisting of many filters more-or- 
less similar to the ones illustrated. If they were located in a wedge-shaped 
region of neural tissue, it might not be unreasonable to suppose that the fil- 
ters at the apex of the wedge, restricted by geometry to have only short 
auxiliary neurons, might respond preferentially to high frequencies, whereas 
those near the base of the wedge, equipped perhaps with multi-neuron feed- 
back chains, might respond preferentially to low frequencies. The notion of 
an array of filters thus seems to me worth holding in mind as a heuristic 
model — worth holding along with other notions, of course — for reference 
during efforts to ,,break” the code of the nervous system. One mighi be alert, 
for example, for any evidence of progressive changes in fiber lengths (or, 
better, in numbers of neurons in intercalated chains), and particularly for ar- 
rays more-or-less at right angles to the dimension of the tissue in which the 
cochlear frequency analysis is represented tonotopically. At the same time, 
he should keep in mind that neuronal arrangements at first glance dissimilar 
to those of Figs. 2 and 3 may be functionally equivalent. The arrangement of 
Fig. 5 (top), for example, is functionally equivalent (except for over-all time 
delay) to the arrangement of Fig. 2. 
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Fig. 4. Frequency-selective neuronal network showing a closer approach than Figs. 
2 and 3 to the behavior of a familiar bandpass filter. The graphs show the excitatory 
processes set up individually (1—7) and collectively (8) by the auxiliary neurons 
in response to a single input impulse. The analogy between the collective curve of 
excitation versus time and the impulse response of a linear network is limited, but 
nonetheless suggestive of kinship between possible neural processes and filtering. 


22 


Downloaded by [Laurentian University] at 22:15 10 April 2016 


It is difficult for me to decide whether notions such as those illustrated in 
Figs. 2, 3 and 5 are ridiculously far-fetched or eminently reasonable. When | 
am inclined toward the former opinion, | may happen upon a schematic dia- 
gram of the lower part of the auditory nervous system or a histological plate 
showing the cochlear nucleus. Then | must ask myself whether the early di- 
vision (in the cochlear nucleus) of the auditory system into two separate 
ascending branches (with preservation of tonotopic localization in both) and 
the later reconvergence (e.g., at the inferior colliculus) have a purpose, and 
whether the gross resemblance to Fig. 4 (bottom) is entirely coincidental. 


B 


QO er 
D E 


D E 


Fig. 5. Another frequency-selective neuronal network functionally equivalent to the 
network of Fig. 2. A single unit is shown at A; an array is shown at B. The essential 
characteristic is that the two paths have different time delays. This characteristic is 
basic, also, to the autocorrelator shown in Fig. 7. 
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Fig. 6. Fourier transformers consisting of correlators fed by oscillators. A shows 
schematically an array in which an input waveform is multiplied by each one of sev- 
eral reference oscillations, after which each product is smoothed (by an integrator 
with negative feedback) to yield a running spectral coefficient. B and C illustrate neu- 
ronal approximations to A. In B, the reference oscillations are supplied by a recurrent 
circuit of neurons. In C, it is assumed that the first neuron in pair fluctuates period- 
ically in sensitivity. 
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Fourier-Analyzer Model 


Another way to effect periodicity-to-place transformations of essentially 
the kind produced by filter banks is to employ an array of correlators fed by 
oscillatory reference signals. Each element of the array must have a refer- 
ence signal source, a multiplier, and an integrator (see Fig 6A). For our pur- 
pose, which is to make a short-time, running correlation, the integrator should 
be ,,leaky”. (The dotted lines in Fig. 6A convert the integrators of that figure 
into smoothing filters.) 

Smoothing is accomplished, in the nervous system, by ,,temporal summa- 
tion” at synapses. Multiplication is approximated by a scheme involving ,,spat- 
ial summation” (27). Periodic pulse trains may be generated by rings of neu- 
rons. A neuronal network to produce an approximate running Fourier trans- 
formation might therefore resemble Fig. 6B. The input signal is ,,multiplied” 
by a reference signal at the first cell body of each ,,correlator’, and the 
product is smoothed by the excitation process at the second cell body. The 
region of the second synapse might better be represented as a very complex 
meshwork of dendrites and/or small cells in order to suggest a smoothing 
time constant of perhaps 25 or 50 milliseconds. 

It is well known that some neurons are ,,spontaneously” active and that 
others, without evident external influence, fluctuate periodically in sensitivity. 
A neuronal Fourier transformer might therefore appear macroscopically to be 
even simpler than Fig. 6B. In Fig. 6C, it is necessary to assume only that the 
first neuron in each horizontal line undergoes periodic fluctuations of thres- 
hold and that the second has a Jong period of temporal summation. The ar- 
rangement of Fig. 6C is then functionally equivalent to the arrangements of 
Fig. 6A and Fig. 6B. 


Autocorrelator Model 


Whereas the filter-bank and Fourier-analyzer models have not, to my know- 
ledge, been proposed as definite auditory hypotheses, a scheme based on 
autocorrelational analysis has been described as a possible periodicity-to- 
place transformer (26, 27, 28). A single hypothetical autocorrelator is shown 
schematically in Fig. 7. 

The purpose of an autocorrelator of the type illustrated is to determine, 
not a single coefficient, but a function. The function itself may progress with 
time: it is a running autocorrelation function. It carries the same information 
about the input waveform as does the running power spectrum — it repre- 
sents everthing except those features of the fine time structure that depend 
upon phase relations among frequency components that are not close neigh- 
bors to one another. The running autocorrelation function is 

plz, t) = F(t) f(t-—z) 
where f(t) is the input waveform, r is a variable shift in time, and the overbar 
designates the taking of a running average. In a neural autocorrelator, the 
time shift r would be introduced by transmission of the signal through neural 
tissue, and it would appear in the output as a shift along a spatial dimension, 
say y. (A more detailed discussion of autocorrelation is given in reference 
26.) 
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Fig. 7. Neuronal autocorrelator. The chain (B) of short neurons (or some other neural 
mechanism providing slow propagation) delivers the input signal to B synapses under 
increasing time delays, whereas the straight-through neuron (A) delivers it without 
significant delay. Spatial summation at B constitutes an approximation to multiplication. 
Temporal summation at C constitutes an approximation to smoothing. Thus the net- 
work produces an array of slowly varying output time functions that approximate the 
running autocorrelation function of the input time function. 


The autocorrelational periodicity-to-place transformer involves an array of 
autocorrelators of the type illustrated in Fig. 7 and thus far discussed. It as- 
sumes that the signals in the various frequency channels of the afferent au- 
ditory pathway — the channels (along what we shall call the x dimension) that 
project the cochlear analysis upward — are fed to individual autocorrelators, 
and that each autocorrelator analyzes the periodicity of neural discharge in 
its particular channel. Each individual autocorrelator converts a single time 
function into a function of one spatial variable (the y dimension of the neural 
tissue) and of time. The array of autocorrelators therefore converts the array 
of neural! time functions (one for each frequency channel along x) into a two- 
dimensional (x, y) manifold of ascending signals, each signal being a function 
of (gross) time. The macroscopic geometry of this process is illustrated in 
Fig. 8. 

The autocorrelator model provides a rationale for several psychophysical 
facts. It provides a mechanism for the residue phenomenon, the pitch of in- 
terrupted random noise, the consonance of octaves, thirds, fourths, etc., and 
even to some extent for certain characteristics of ,,absolute” pitch judgments. 
These congruences between model and observation have been described 
(26-28). It seems unwarranted to redescribe them here. It may be worth 
noting, however, that two psychoacoustic phenomena not heretofore discussed 
in connection with the autocorrelation model are consonant with it. First, the 
sweep pitch” described by Thurlow and Small (43) appears to arise when, 
and only when, there is a swath of the autocorrelation surface with periodic 
ridges at those values of + (or y) at which ridges appear in response to a 
sinusoidal tone of the same pitch as the sweep pitch. Second, the ambiguity 
of the pitch of the residue set up by certain pulse patterns in experiments by 
Flanagan and Guttman (11) is consistent with the fact that different swaths 
of the autocorrelation surface may at a given moment have periodic ridge 
patterns with different periods, patterns corresponding to different low-fre- 
quency tones. 
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The autocorrelator mode! makes assumptions about the functional and topo- 
logical (and to a limited extent the metric) interrelations of neurons in the 
auditory system, and it implies predictions about recordable electrophysio- 
logical events. At some points, the model can be held up against histological 
and electrophysiological facts, but there is available, to the best of my know- 
ledge, no crucial confrontation. 

The branching and reconvergence of the ascending auditory pathways, men- 
tioned in connection with the filter-bank model, provides a gross structure as 
suitable for an autocorrelator as for an array of filters. The range of latencies 
observed in electrical responses of cells in the medial geniculate body (10 
to 125 milliseconds) (16, 17, 42) indicates that there is easily enough relative 
time delay in the auditory tract to meet the design requirements of an auto- 
correlator. The midbrain, thalamic, and cortical auditory centers abound in 
fine structure suitable for either model. However, | know of no sharply indi- 
cative neuroanatomical evidence, and am therefore left with the feeling that 
the anatomical assumptions of the autocorrelation model are merely plausible 
— neither rejected nor strongly supported. 

The electrophysiological side of the picture is only a little more definite. 
All three classes of models that we have considered imply that there is a 
two-dimensional tonotopic projection into one or more of the higher auditory 
centers, and that, if the meaningful reference frame of that projection is simply 


Fig. 8. Array of neuronal autocorrelators. Each horizontal subsystem corresponds to 
an autocorrelator of the type illustrated in Fig. 7. The array of autocorrelators performs 
a periodicity analysis upon the array of cochlear outputs, yielding a two-dimensional 
display of the activity that underlies subjective pitch. 
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a two-dimensional surface or two dimensions of a region of the neural tissue, 
responses to periodic high-frequency (residue-producing) signals should be 
recordable from somewhat different places than those responsive to aper- 
iodic high-frequency (non-residue-producing) signals. Kiang and Goldstein 
{24) applied various periodic and aperiodic waveforms to the ear and re- 
corded electrical potentials from strychninized spots on the auditory cortex 
of the anesthetized cat. They found no evidence at all of periodicity localiz- 
ation of the kind implied by the models. However, the cortex may not be the 
place to look for correlates of pitch, and the strychnine spike response (us- 
ually a transient response to transient stimulation) may not be the kind of 
indicator with which to search for a neural process underlying a sustained 
(pitch) experience. As Kiang and Goldstein say, the electrophysiological re- 
sults do not argue for a pure place theory of pitch perception. The results 
seem to me to be slightly negative, but only a little below neutral, insofar as 
the periodicity-to-place conversion hypothesis is concerned. 


Unordered-Array Models 


Perhaps the least plausible assumption made in formulating the models thus 
far discussed is the assumption of ordering in the arrays of filters, oscillators, 
and delay channels. In proposing the autocorrelator model in the ,,duplex 
theory”, | stressed that the orderliness and regularity of arrangement shown 
in the schematic representations were introduced only for the sake of con- 
ceptual simplicity and might not be found in the nervous system (26). In dis- 
cussing the ,,triplex’’ version of the theory (an extension designed to account 
for the Huggins-Cramer phenomenon), | noted that a random neuronal net- 
work would perform many of the operations performed by the deliberately 
organized networks (28). Here let us examine the implications of removing 
entirely the assumption of ordering in the arrangement of the arrays involved 
in periodicity-to-pitch conversion. (The order of elements in the x dimension, 
arising from the cochlear frequency analysis, is of course retained.) The 
assumptions involved in the three classes of models are reduced, by this eli- 
mination, to specifications that there exist, in the neural tissue corresponding 
to each interval along the x dimension, several of the following: 


1. delay elements capable of introducing assorted values of 7; 

2. coincidence-sensitive synapses; 

3. synapses or networks capable of temporal summation; 

4. recurrent neuronal circuits or other devices capable of producing more- 
or-less sustained oscillation; 

5. inhibitory as well as excitatory influences; and 

6. spontaneous periodic fluctuation of sensitivity. 


Those assumptions correspond to standard, or at least to demonstrated, 
characteristics of the behavior of central neurons, synapses, and nets. It 
seems very unlikely, therefore, that a haphazard network of many thousand 
neurons would fail to approximate, in one small zone or another, filtering 
and correlation with a reference oscillation and correlation with the delayed 
input signal, itself. Given tens or hundreds of thousands of neurons in quasi- 
random configuration, one might find band-pass filtering approximated with 
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several different center frequencies and bandwidths, correlation approximated 
with several reference-oscillation frequencies and smoothing time-constants, 
etc. If the ordering of arrays is not crucial, that is to say, one might argue 
that the auditory nervous system can hardly avoid incorporating all three 
kinds of model, and other permutations of the basic operations as well. 

Of what consequence to the nervous system, then, would be the ordering 
of similar component processes into graded sequences following spatial di- 
mensions in the tissue? Why would a designer introduce such ordering? If 
it came as a by-product of some other effort, if it were simply easier to build 
an ordered frequency analyzer (e.g., a cochlea) than a haphazard one, then 
well enough — but is there any intrinsic advantage in achieving the order 
of the graded sequence that should Jead one to postulate such order in a 
model of the kind with which we are here concerned? 

It seems to me that a fundamental place principle of neural processing is 
collocation for correlation: bringing together in one place those subprocesses 
that must be interrelated and integrated to represent a unitary concept or 
object or to carry out a coherent act. If the main business of evolving orga- 
nisms had been to serve as subjects in experiments on the missing funda- 
mental, then, according to the principle, periodicity pitches would be lined up 
next to one another, in regular ascending order, as suggested by the dia- 
grams of models. On the other hand, if the main business involved the iden- 
tification of diverse objects partly on the basis of their sounds, then the 
pitches made by particular objects or classes of objects would be separated 
from other pitches and coupled closely to characteristic timbres and volumes 
and shapes and textures. This line of thinking leads me to devalue (though 
not entirely) the hypothesis that there is order in what we have been referring 
to as the y dimension. Order and sequence are present at the outset in the x 
dimension, thanks to the cochlear analysis. It seems likely, however, that 
nature probably leaves the y dimension in disorder — that the preferred mod- 
el for the untrained organism is a model involving periodicity-to-place con- 
versions that assign the various periodicity pitches to more-or-less randomly 
selected places in a nucleus of the auditory system. Nature, then, may 
shape up whatever arrangements and organizations are required to meet the 
demands of individual experience. 

A discussion of the ,,triplex theory’ contains some speculations about a 
process through which an organized network might be developed by train- 
ing and experience (28). However, | think much experimentation with simulated 
neuronal networks will be required to develop an understanding of such a 
process. Let me therefore go no further on that path now. 


Property Filters 


The idea that elementary processing operations may be arranged in quasi- 
random and therefore diverse configurations in the auditory system tends to 
open the domain of speculation beyond frequency and periodicity analyzers. 
One envisions an aggregation of processing units — filters in a very gener- 
alized sense — each one selectively responsive to one or more spatio-tem- 
poral patterns of incident excitation. Some of these units might approximate 
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paradigms of the type we have discussed and respond selectively to par- 
ticular frequencies. Others might respond selectively to particular distribu- 
tions inthe x dimension of their millieux, and thus to particular acoustic energy 
spectra. Still others might respond to particular binaural coincidences (see 
below) or to excitation patterns set up by clicks, trills, glissandi, etc. Each 
unit would ,,recognize” the presence of a particular property in the neural 
excitation reaching it — which is almost to say, a particular property of the 
stimulus. 

Galambos (15) has described electrophysiological observations that in- 
dicate that some neurons in the auditory cortex (and some in the medial ge- 
niculate body) of the cat do behave in the way just described. Mountcastle 
(30) has found property filters in the spinal somesthetic system of the cat. 
Letvin et al. (25) have found them in the optic nerve of the frog. Evidently, 
property filters are fundamental building blocks of neural processes. In re- 
search in the new field of ,,artificial intelligence’, property lists are among 
the most sophisticated techniques of pattern recognition, and property filters 
are employed in several of the most successful pattern-recognition systems 
(29). Under the influence of these considerations, | find myself tending to 
recast hypothetical auditory mechanisms in such a way as to emphasize the 
kinds of analysis that lead to synthesis (pattern recognition, object construct- 
ion, concept formation, problem solving) as opposed to the kinds of analysis 
that lead to scales that correspond to sensory attributes. This tendency does 
not deny the scalability of sensory attributes, nor the primary nature of such 
attributes as the one based upon the cochlear frequency analysis. It does, 
however, de-emphasize the need for ordered sequences, and it emphasizes 
the importance of variety in the arrangement of the elements of neural pro- 
cessing systems. 


SHARPENING, FUNNELING, SEPARATION AND FUSION, 
AND BINAURAL INTERACTION 


Models designed to account for periodicity pitch should be consistent with 
models designed to account for other major auditory phenomena. ldeally, 
models of various subordinate processes should fit together into one coherent 
theory of hearing. Although it is not possible to advance very far toward that 
ideal with the parts presently available, something should be said about some 
of the other processes that are closely associated with the hypothetical pro- 
cesses underlying the periodicity-to-place transformation. 


Sharpening 


As mentioned earlier, the cochlear frequency analysis is not very sharp. 
Much of the basilar membrane is set into vibration even by pure-tone sti- 
mulation. Successive stages of the auditory process, all the way up to the 
medial geniculate body, concern themselves, in part at least, with the task 
of sharpening further the distributions of activity that ascend the auditory 
pathways in response to frequency-structured sounds (16, 42). Is the fine 
time pattern of the neural signals involved in the sharpening process? 
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Periodicity analysis of the general type we have been discussing serves 
some of the same purposes as sharpening of the peaks in the x distribution 
of neural activity. Periodicity pitch extracts important information from that 
part of the spectrum in which the cochlear frequency analysis is least sharp. 
lt displays that information in a convenient way, one that changes progressiv- 
ely, as the period increases, from a continuous pitchlike quality through gra- 
dations of increasing roughness until, as the analysis breaks down, the sound 
becomes sensibly intermittent and the stimulus time structure is perceived 
as a succession of events in phenomenal time. This way of handling fine tem- 
poral detail seems preferable to straightforward augmentation of resolution 
in frequency, which would be difficult to blend into perception of gross time. 

There appears to be no clear evidence that the fine time or frequency 
structure of the neural signals plays any role, beyond that associated with 
periodicity pitch, in the sharpening process. In a paper on place mechanisms 
of frequency analysis, Huggins and | (22) once said that we planned a sequel 
on time-domain mechanisms; the sequel did not materialize, mainly for want 
of ideas beyond those relating to periodicity pitch. 

Hypothetical place mechanisms of sharpening depend mainly upon (1) 
special lateral extensions or coursings of sensory neurons, (2) Jateral inter- 
connections in central nerve nets, or (3) simultaneous convergence and diver- 
gence in projection systems. One of the basic principles is that focused, 
sharply localized excitatory influences combined with diffuse, spreading in- 
hibitory influences, produce sharpening of peaks and contours (22). There is 
no evident incompatibility between arrangements of the kind required for 
sharpening and those hypothesized for periodicity-to-pitch conversion. The 
main obstacle to bringing the two kinds of paradigm together in one model 
is that it is very difficult to incorporate so much detail into a comprehensible 
verbal and pictorial description. The main hope appears to lie in simulation 
through digital computer programs. 


Funneling 


Békésy (4-6) has defined and used repeatedly a principle related to shar- 
pening but incorporating an additional feature. It is convenient to describe 
it by referring to a somesthetic phenomenon that he observed (4). When force 
is applied steadily to a long, narrow area of his skin, it gives rise for a pro- 
longed period to a correspondingly long, narrow image. When, however, the 
force is applied suddenly, it feels as though the blow were confined to a short 
segment near the middle of the stimulated strip — but the overall strength 
of the sensation is not correspondingly diminished. It is as though all the 
stimulus energy were poured through a sensory funnel into the short, central 
segment. 

There appear to be two main approaches toward development of a model 
of the underlying process. The term ,,funneling” is more appropriate to the 
first than to the second. 


(1) In keeping with the basic idea of funneling, one might devise a neuron- 
al network in which peripheral parts of an ascending (or in-flowing) pattern 
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of neural activity progressively decline in strength, while central parts increase 
in strength. The lateral extent of the pattern would decrease, but the total 
amount of activity would remain approximately constant. Such a mechanism 
can be constructed with schematized neurons having excitatory and inhibitory 
interconnections. 


(2) In keeping with the idea of property filters, on the other hand, one might 
set up an arrangement in which various neural units measure (respond se- 
lectively to) diverse aspects of the peripheral excitation pattern: its strengths 
in various parts of the area, its gradients, its curvatures, etc. Other neural 
units, perhaps at a higher level, would respond to signals from combinations 
of peripheral units. The responses of some of the latter would be diagnostic 
with respect to ,,where’, others to ,,how much”, still others to ,,what shape”, 
etc. The perceptual mechanism would then synthesize a sensation by using 
the diagnostic responses as data for a decision process. In such a model, 
the gross pattern of neural activity might or might not actually converge as 
though funneled, and the total amount of activity might remain constant, de- 
crease, or increase. As soon as the information is encoded (presumably in 
the ,,place’’ of the measuring unit or property filter), the amount of activity 
becomes less important as a variable. 

It is necessary, of course, to design into the second type of model, just 
as into the first, the basis for the ,,mistake” in the judgment of the spatial 
distribution of the sudden blow. However, the necessary features could be 
designed into the extent-measuring part of the system without having to affect 
materially the parts responsible for determining such other characteristics as 
the center coordinates and the total force of the blow. 

The foregoing considerations apply also, of course, to hearing. The second 
approach is highly compatible with the unordered periodicity-to-place models. 
It leads one to think of measuring separately the pitches and loudnesses of 
the several parts of a compound signal, and then synthesizing the sensation 
from the measured data. 


Separation and Fusion 


What are the ,,parts’’ of a compound signal? How are the superposed 
sounds of several voices or instruments, for example, separated in analysis 
and assigned individual pitches, timbres, loudness, etc.? Or, to look at the 
opposite face of the coin, how can certain parts of a spectrum be brought 
together to form one unitary sound, while other parts form a second unitary 
sound, and still other parts make up a distributed noise? As Sayre and 
Cherry (33) have emphasized, the problem of separation and fusion is fun- 
damental. It is fundamental to pitch perception, because the several parts of 
a compound auditory sensation may have distinct pitches. It is fundamental 
to binaural hearing, because approximately paired parts of a dichotic signal 
fuse together while other parts fail to fuse and are localized separately near 
the listener's ears. 

It is beyond the scope of this paper to attempt to examine fusion hypo- 
theses exhaustively. Let us limit ourselves, therefore, to a few ideas con- 
cerning fusion that have been related also to periodicity pitch. 
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Binaural Interaction 


In trying to extend the ,,duplex theory’ (autocorrelator model) in such a 
way as to account for the Huggins-Cramer phenomenon, | borrowed a notion 
from Wallach (44) and Jeffress (23) that can be called, with some appropriate- 
ness, a cross-correlator model. | shall not describe its action here. The pres- 
ently relevant point is that it seemed necessary to have the mechanism 
of binaural interaction (i.e., the cross-correlator) operate rather directly upon 
the cochlear output in order, for example, to separate the homophasic from 
the heterophasic parts of the Huggins-Cramer noise and at the same time 
fuse the left and right homophasic parts into one unit and the left and right 
heterophasic parts into another unit. The cross-correlator was assigned a 
short smoothing time-constant, and its output could be used as the input sig- 
nal for the periodicity-to-place mechanism (the autocorrelator). If the auto- 
correlator, which had to have a longer time-constant, had been placed first 
in a tandem configuration, the cross-correlator would not have had a reason- 
able input signal. 


As | see it now, it would have been better to adopt the property-filter ap- 
proach, to have regarded the periodicity measurements and the interaural- 
time-delay measurements as measurements of separate properties that could 
just as well be determined in parallel. The periodicity-to-place transformation 
mechanism should probably be associated with an extended segment of the 
afferent pathways, starting at the cochlear nucleus, whereas the binaural- 
interaction mechanism might well be associated with a part of one of the 
pathways that includes the olivary complex (18, 40). The binaural-interaction 
mechanism should of course be up-dated by talking into account, as Van 
Bergeijk (1) has done, the new anatomical and physiological information (18, 
40) about the olivary body which reveals its accessory nucleus as a highly 
specialized arrangement obviously designed to handle interaural time delays. 
Neurons of the accessory nucleus have two dendrites, one reaching out to- 
ward the left ear and the other reaching out toward the right, and firing is 
inhibited by certain temporal relations between excitations applied by the 
two dendrites. The existence of this so specialized component in the auditory 
system lends support to the idea that, whereas the general features of audi- 
tory processing are similar to those of the skin senses (4-6), a considerable 
amount of ad hoe design is defensible in formulating models of auditory 
processes. 


After modification to take into account the new information, the model for 
binaural interaction looks somewhat less like a cross-correlator than it did 
before, but is clearly behaves like a cross-correlator. That is evident in the 
results of Cherry and Sayres, who demonstrated remarkable correspondence 
between the responses of a cross-correlator model and the laterality judg- 
ments of subjects listening to a variety of dichotic sounds (7). 

The property-filter approach (but not the notion of unordered arrays) ap- 
pears tobe applicable evento the aspect op pitch that is based on the coch- 
lear analysis and represented in the x dimension of the tissue. With that 
approach, triplex theory could account for a theoretically crucial phenome- 
non observed by Franssen (14). Franssen listened to an ingeniously contrived 
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dichotic signal consisting of two pulses of oscillation. The pulse delivered to 
the left ear led in time and oscillated at frequency f,, but it was short and 
(heard alone) almost devoid of pitch. The pulse delivered to the right ear, 
which followed in time, oscillated at frequency f,, well separated from f,, and 
it was long enough to have a definite pitch. When the two pulses were pre- 
sented together, one to one ear and the other to the other, the listener heard 
a single, fused sound. It appeared to be located near his left ear, but its 
pitch corresponded to f,. Evidently, the auditory system handled the ques- 
tions of pitch and localization separately and made an ,,error” in the process 
of synthesizing a sensation to explain the data economically. 


»PERIODICITY PITCH” ET MODELES ALLIES DE 
PROCEDES AUDITIFS 


Le phénoméne résidu de Schouten (34-38) contredit clairement la théorie de 
location (place theory) de la perception de la période subjective (pitch), car le 
résidu posséde une période subjective qui n’est pas propre aux canaux de la 
fréquence Fourier du systéme auditif par laquelle elle est entendue. Quand, 
par exemple, un stimulant ondulatoire (voir Fig. 1A) (qui ne contient pas 
d’énergie 4 basse fréquence) se trouve en face d'un fond intense de bruit a 
basse fréquence, l’auditeur entend un ton bas (le résidu), bien qu'on puisse 
supposer que seuls sont activés les canaux inférieurs de haut fréquence du 
nerf auditif (27, 28). (Quand une sinussoide 4 basse fréquence se trouve en 
presence du méme bruit de fond, aucun son n’est entendu; le signal de basse 
fréquence est entigrement masqué par le bruit.) Evidemment quelque revision 
de la théorie de location pure est nécessaire pour cette raison. 

La théorie de location est si efficace pour [’explication d'autres phéno- 
meénes auditifs et si bien en accord avec des observations neurophysiolo- 
giques, qu'on ne peut penser a la rejeter et 4 remplacer. Au lieu de cela il 
parait approprié de |'étendre pour donner un mécanisme qui obéit la pério- 
dicité dans les décharges de neurones (ou de groupes de neurones) dans des 
centres bas du systéme auditif et qu'une telle périodicité est converté en lo- 
cation dans des centres plus hauts. Un tel mécanisme pourait offrir une ex- 
plication pour le résidu de période (residue pitch), !a période subjective du 
bruit blanc interrompu, la consonance musicale, et d'autres phénomeénes (26). 

ll y a trois sortes de mécanisme général qui s’offrent pour étendre la théo- 
rie de location. Elles sont typifiées par 1.) le modéle des groupes de filtres 
(voir Figs. 2-5), 2.) le modéle de l’analyseur de Fourier (voir Fig. 6), et 3.) le 
modéle de l’autocorrélateur (voir Figs. 7 et 8). Il apparait possible que le 
systéme nerveux, quand il emploie seulement les opérations établies et élé- 
mentaires du dilai de temps, le sommation dans l'espace, la sommation dans 
le temps, recurrent-circuit feedback, |’excitation, et l’inhibition pour exécuter 
les fonctions qui se trouvent dans les trois modéles. 
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